The same issue was raised again a few years later by R. Slifkin in his discussion of the Kezayis. In his essay "The Evolution of the Olive", R. Slifkin asserts, based on historical evidence which includes testimony of Rishonim themselves, that Ashkenazic Rishonim generally never saw an olive in their lives.
Recently I came across another such instance where an Ashkenazic Rishon is accused of erring due to not having seen the subject-matter at hand. R. Yaakov son of the Rosh writes in his monumental work "Arba'ah Turim" (Tur OC §202) that since sugarcanes are mostly grown to extract their juice for sugar production but not for eating the cane itself because that is inedible and therefore the appropriate brakhah on sugar would be Bore Pri Ha'etz. But R. Yosef Karo in his Kesef Mishna (Brakhot 8:5) comments on the Tur's words - and the Magen Avraham (OC §202:13) cites the words of the Kesef Mishna approvingly - as follows:
ואני אומר שאילו היו קנים הללו נמצאים בארצו של הטור לא היה טוען כן. שבמקום שנמצאים מוכרים מהם לאלפים ולרבבות למצוץ אותם.
And I say, were these canes sold in the land of the Tur, he wouldn't have said so. Because in the places where they are found, they are sold in the thousands and ten thousand to be sucked.
Sugarcanes are usually grown in tropical regions (as per Wikipedia) The Tur claimed that the canes are inedible, but R. Karo says (as does -l'havdil- Wikipedia) that in the places where sugarcanes are grown they are regularly eaten. The Tur just never saw a sugarcane in his life.