For once, it's simple.
The Japanese are descendants of two sets of ancestors: An aboriginal population (survivors of which are the Ainu of Hokkaido) and the Yayoi culture people who had recently immigrated from southern Korea at the beginning of the study period. Although the two groups surely intermarried, the Japanese culture as we know it was developed by descendants of the Yayoi people. If your ancestors were in Japan in CE 500, then their ancestors were in both Japan (as Ainu) and southern Korea in CE 1. Deal with it.
Note also: Japanese culture spread from south to north. During the Middle Ages, places like Tokyo were at the northern fringe of "civilization." Even as late as the 19th century, Hokkaido was considered a wild frontier. So, if your ancestors are from Northern Japan, it's safe to assume that many of their ancestors came from farther south in earlier periods.