Born in Riga, he was educated at the commerce school of the Riga Stock Exchange, and graduated in Moscow. In 1917, he began studies at the Riga Polytechnical Institute which had been evacuated to Moscow. After the October Revolution, he returned to Latvia.
In 1918, Celmiņš enlisted into the newly-created Latvian Army, and was promoted to lieutenant the following year, and was then appointed Latvian military attaché in Poland. In 1921, he was awarded the Order of Lāčplēsis.
Retired from army in 1924, he worked in the Ministry of Foreign Affairs from 1925 to 1927. Celmiņš became the secretary of Minister of Foreign Affairs, and subsequently worked in the Finance Ministry. On 24 January 1932, the Latvian nationalist group Ugunskrusts was founded, and Gustavs Celmiņš was elected as its leader. After Ugunskrusts was banned, he founded the organization Pērkonkrusts ("Thundercross"). Common for both organisations was that they advocated a national revolution for a radical re-organisation of society, politics, and the economy in Latvia. Following Kārlis Ulmanis' 15 May 1934 coup d'état, Celmiņš was arrested and imprisoned for three years. He was exiled from Latvia in 1937.
Celmiņš moved to Italy, then Switzerland. While in Zürich, he was arrested and then banished from Switzerland. He later lived in Romania, where he had contacts with the Iron Guard, and then moved to Finland. In 1938, he became the leader of Pērkonkrusts "foreign contacts office". After the Soviet Union invaded Finland, Celmiņš enrolled as a volunteer on the latter's side. When the conflict ended, he moved to Nazi Germany.
After the occupation authorities once again banned Pērkonkrusts in August 1941, Celmiņš continued his outward collaboration with the Germans in the hopes that sizable Latvian military formations would be created. From February 1942, he headed the Committee for Organising Latvian Volunteers (Latvian: Latviešu brīvprātīgo organizācijas komiteja), the main function of which was the recruitment of Latvian men for the Latvian Auxiliary Police Battalions, known in German as Schutzmannschaften or simply Schuma.  Aside from front-line combat duties, these battalions were also deployed in so-called anti-partisan operations Latvia and Belarus that included the massacres of rural Jews and other civilians. This situation was not what Celmiņš had hoped for, and so he began to sabotage the recruitment efforts. Because of this, he was later transferred to a job as a minor clerk within the occupation administration.
Pērkonkrusts members working within the SD apparatus in occupied Latvia would feed Celmiņš information, some of which he would include in his underground, anti-German publication Brīvā Latvija. This eventually led to Celmiņš and his associates being arrested by the Gestapo in 1944, with Celmiņš ending up imprisoned in Flossenbürg concentration camp.
In late April 1945 he was, together with other prominent concentration camp inmates, transferred to Tyrol where the SS left the prisoners behind. He was liberated by the Fifth U.S. Army on 5 May 1945.
After World War II, he lived in Italy, where he published the newspaper Brīvā Latvija. In 1947 he published the autobiographic book Eiropas krustceļos ("At the Crossroads of Europe").
In 1949 he emigrated to the United States. From 1950 to 1952 he was an instructor at Syracuse University's Armed Forces school in New York state, and beginning in 1951 he was also the director of the Foreign Language program for the US Air Force, and a television lecturer about the USSR and communism. From 1954 to 1956 he worked as a manufacturer in Mexico. Between 1956 and 1958 he was a librarian at Trinity University in San Antonio, Texas. In 1959 he became a professor of Russian studies at St. Mary's University in San Antonio, Texas. He died on 10 April 1968 in San Antonio, Texas.