Continuous-variable quantum key distribution (CV QKD) has many practical advantages including compatibility with current optical communication technology. Implementation using heterodyne measurements is particularly attractive since it eliminates the need for active phase locking of the remote pair of local oscillators, but the full security of CV QKD with discrete modulation was only proved for a protocol using homodyne measurements. Here we propose an all-heterodyne CV-QKD protocol with binary modulation and prove its security against general attacks in the finite-key regime. Although replacing a homodyne measurement with a heterodyne measurement would be naively expected to incur a 3 dB penalty in the rate-distance curve, our proof achieves a key rate with only a 1dB penalty.