How much does sector matter?

Hi Monkies,

I have to decide between starting entry level in a new sector or staying at my current level in the same sector and moving banks (I'm currently at GS/MS/JPM). I do want to explore other sectors, but I also don't want to sacrifice years of my career if sector doesn't matter that much. Let's say for both choices I would be working with a highly ranked analyst. Does sector matter that much? Are there any to avoid? Are there any that develop less transferable skills than others? For example FIG?

Thanks!