This is the question that I have asked most people in interview (including the engineers who claimed to know about specman). I didnt get satisfactory answer.
If I have to give analogy of a computer, think of 'unit' like a hardware and 'struct' like a software. Unit has to be tied to a physical entity and solely dedicated to that physical entity (like a port) . For example, you will unit to code your BFMs and Collectors. However any item say a 'packet' or 'frame' which you need to generate has to be 'struct'.
'Struct' isn't tied to any physical entity. any component can use the struct. For example, same packet 'struct' is used by many Port (BFMs and Collctors) to process/send/receive the packet or frame.
Lot of people give reasons like they will use 'unit' because they need 'tcm'( or notion of time). However this is only secondary reason behind using unit. Primary reason for using 'unit' should be whether the functionality which you want to implement is tied to physical entity or not.