Joseph (Imhotep) had long since died and the Pharaohs of the 12th dynasty had forgotten his connection to the Israelites. Joseph / Imhotep had looked after his family and ensured that they were able to multiply to become a great nation. Joseph died at the age of 110 years; 80 years after coming to Egypt.
The Israelites were in Egypt for 430 years by the time of the Exodus.
The Israelites had become so numerous by the start of the 12th dynasty that the Egyptians were beginning to feel threatened.
Amenmhet I (the first pharaoh of the 12th dynasty), was the vizier of Mentuhotep IV (the last pharaoh of the 11th dynasty) whom he may have assasinated.
The pharaohs of the 12th dynasty required a large slave labour force for the construction of their pyramids and other public works. The Jews were given the task of making mud bricks and working in the fields. The 12th dynasty lasted almost 200 years and at least seven pyramids were constructed during this time.
A semitic slave village at Kahun was built during the time of Sesostris II and occupied up until the time of the Exodus when Neferhotep I was reigning. We know this because scarabs of these pharaohs were found in the town.
The pyramids of the 12th dynasty had a core of mudbricks and a limestone veneer. The cores of each of these pyramids contained millions of mudbricks. Mudbricks were also used for other structures such as the Labyrinth which was built by Amenemhet III, the 6th pharaoh of the 12th dynasty.
Amenemhet III reigned for 46 yrs but had a 20 year co-regency with his father Sesostris III at the beginning of his reign. Moses was born early in the reign of Amenemhet III. It was Amenemhet III who tried to kill the male Israelite babies at birth. It was his daughter Sobekneferu who found Moses in a basket among the reeds of the Nile and adopted him and raised him as her own. Moses was groomed to be the next pharaoh Amenemhet IV and may have even been co-regent with Amenemhet III for 9 years (as Amenemhet IV). He never got to rule by himself. When his affections for the Israelites became known, Moses (alias Amenemhet IV) had to go into exile in Midian or be killed by Amenemhet III.
This not only fits well with historical records and the archaeological evidence, it allows complete synchrony of the Bible with the Egyptian Chronology.