Skip to content

Use UTF-8 encoding when reading pyproject.toml#588

Merged
messense merged 1 commit intoPyO3:mainfrom
domdfcoding:toml-utf8
Jul 12, 2021
Merged

Use UTF-8 encoding when reading pyproject.toml#588
messense merged 1 commit intoPyO3:mainfrom
domdfcoding:toml-utf8

Conversation

@domdfcoding
Copy link
Contributor

@domdfcoding domdfcoding commented Jul 11, 2021

This avoids a UnicodeDecodeError when reading pyproject.toml files containing unicode characters on Windows. These characters may be present in a project's metadata, such as its description or an author's name.

>>> from maturin import get_config
>>> get_config()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib\site-packages\maturin\__init__.py", line 39, in get_config
    pyproject_toml = toml.load(fp)
  File "lib\site-packages\toml\decoder.py", line 156, in load
    return loads(f.read(), _dict, decoder)
  File "lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 242: character maps to <undefined>

Furthermore, the TOML specification requires files to be encoded in UTF-8.

messense
messense approved these changes Jul 12, 2021
View reviewed changes
Copy link
Member

@messense messense left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@messense messense merged commit b992282 into PyO3:main Jul 12, 2021
messense added a commit that referenced this pull request Jul 13, 2021
@messense
Add changelog for #588
61f2ff3
bmwiedemann pushed a commit to bmwiedemann/openSUSE that referenced this pull request Aug 3, 2021
@bmwiedemann
Update python-maturin to version 0.11.2 / rev 5 via SR 909889
ee928b2 
https://build.opensuse.org/request/show/909889
by user mcepl + dimstar_suse
- Update to 0.11.2
  * Use UTF-8 encoding when reading `pyproject.toml`
    gh#PyO3/maturin#588
  * Use Cargo's `repository` field as `Source Code` in project URL
    gh#PyO3/maturin#590
  * Fold long header fields in Python metadata
    gh#PyO3/maturin#594
  * Fix `maturin develop` for PyPy on Unix
    gh#PyO3/maturin#596
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@messense messense messense approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@domdfcoding @messense