Failing with sqlite3.OperationalError: duplicate column name

[omnidepp]

Hey,
interesting project! I tried to give it a spin, but the mbox file I'm trying to analyze throws me this error:

Traceback (most recent call last):
  File "/usr/local/bin/mbox-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/mbox_to_sqlite/cli.py", line 33, in mbox
    db[table].upsert_all(to_insert(), alter=True, pk="Message-ID")
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 3183, in upsert_all
    return self.insert_all(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 3082, in insert_all
    self.create(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1574, in create
    self.db.create_table(
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 974, in create_table
    self.execute(sql)
  File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 485, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: duplicate column name: Message-Id

[Robert-Ernst]

Same issue here

$ mbox-to-sqlite mbox emails.db INBOX
Traceback (most recent call last):
  File "/home/uesrname/.local/bin/mbox-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/uesrname/.local/lib/python3.10/site-packages/mbox_to_sqlite/cli.py", line 33, in mbox
    db[table].upsert_all(to_insert(), alter=True, pk="Message-ID")
  File "/home/uesrname/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 3183, in upsert_all
    return self.insert_all(
  File "/home/uesrname/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 3082, in insert_all
    self.create(
  File "/home/uesrname/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 1574, in create
    self.db.create_table(
  File "/home/uesrname/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 974, in create_table
    self.execute(sql)
  File "/home/uesrname/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 485, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: duplicate column name: Mime-Version

[Robert-Ernst]

How is the status @simonw ?

[McPolemic]

I ran into this problem as well. It looks like Gmail outputs multiple columns that are the same name and just different capitalization. Unfortunately, SQLite column names are case-insensitive. I output the SQL command that mbox-to-sqlite was using to create the table and put it in a gist in case that ends up helping.

https://gist.github.com/McPolemic/5ac8b4bf4d5668c86b21cd92a7eb2488

[shakfu]

Same here.. I got: sqlite3.OperationalError: duplicate column name: Mime-Version

from @McPolemic's gist, it looks there are two columns which are causing the issue in my case: [MIME-Version] TEXT and [Mime-Version] TEXT