“Too many points” when using gdal2tiles.py in EPSG:4326

I want to put a custom map onto cartodb, which, I believes, requires EPSG 4326. When I try making tiles with gdal2tiles, using a 4326 georeferenced image, I get the following error:

ERROR 1: Too many points (10201 out of 10201) failed to transform,
unable to compute output bounds.

If I try to use other projections, gdal2tiles will work fine but they show up in the completely wrong place on cartodb.