Hello @ghuname, welcome to PyScaffold's issues page, nice to see that you are using it 😄

So what is happening is related on how Python's module system and PYTHONPATH work.
Folders are meant to be considered packages, while files are simple modules.

When you install a wheel/sdist package that uses the src-directory structure (as generated by PyScaffold), it is roughly like adding the src directory to the PYTHONPATH (with the difference that you can only have either one .py file or a single folder under src).

There is a trick however: for Python's import system name.py is basically equivalent to name/__init__.py. So the short answer for your question is: if you want to do import scaffmodule; scaffmodule.print_something(), you need to define the function print_something inside the __init__.py file. (Alternatively you can also have the functions/classes that you want to "export" as your public API imported into your __init__.py from submodules, as the famous package click does).

I don't use PyCharm, but I suppose the only reason you are able to do import src.scaffmodule.scaffmodule in its terminal is a side effect of running the terminal from your project's root folder (Python always add the current working dir to PYTHONPATH). This is somehow an unreliable side-effect. What I would suggest is to install your package in the editable mode inside the virtual environment being used by PyCharm with pip install -e . , then you should always be able to do import scaffmodule (without the src part). Just remember to re-install the package (with pip install -e .) everytime you edit setup.cfg.

In case you are interested, there is a bit more in-depth information about Python's module system in the following references:

• Python docs
• Real Python

Please let us know if this solves your problem.

Hi @abravalheri

Hello @ghuname, welcome to PyScaffold's issues page, nice to see that you are using it 😄

Why shouldn't I. PyScaffold is a quite good tool. I really helps in packaging the python code.

So what is happening is related on how Python's module system and PYTHONPATH work.
Folders are meant to be considered packages, while files are simple modules.

When you install a wheel/sdist package that uses the src-directory structure (as generated by PyScaffold), it is roughly like adding the src directory to the PYTHONPATH (with the difference that you can only have either one .py file or a single folder under src).

Do you want to say that if I delete src/scaffmodule folder and put scaffmodule.py directly into the src folder it will work with no double import (import scaffmodule.scaffmodule)?

There is a trick however: for Python's import system name.py is basically equivalent to name/__init__.py. So the short answer for your question is: if you want to do import scaffmodule; scaffmodule.print_something(), you need to define the function print_something inside the __init__.py file. (Alternatively you can also have the functions/classes that you want to "export" as your public API imported into your __init__.py from submodules, as the famous package click does).

scaffold.py is jusst an example. My real code has the following structure:

set some variables
define some function/classes
if __name__ == '__main__':
    parse argparse parameters
    call functions/instance objects

Would it be OK, if I move all the code from scaffmodule.py to src/scaffmodule/__init__.py?
In this case, will I be able to use command line parameters like: python scaffmodule.py arg1 arg2... ?
Do I need to create a shell script in order to call the module directly (if I don't use it as a library)?

I don't use PyCharm, but I suppose the only reason you are able to do import src.scaffmodule.scaffmodule in its terminal is a side effect of running the terminal from your project's root folder (Python always add the current working dir to PYTHONPATH). This is somehow an unreliable side-effect. What I would suggest is to install your package in the editable mode inside the virtual environment being used by PyCharm with pip install -e . , then you should always be able to do import scaffmodule (without the src part). Just remember to re-install the package (with pip install -e .) everytime you edit setup.cfg.

Locally I installed scaffmodule package as: python setup.py develop from scaffmodule root folder.
What bothers me is installation on other systems and at the same time ability to use it in its simplest way as import scaffmodule; scaffmodule.print_something().

In case you are interested, there is a bit more in-depth information about Python's module system in the following references:

• Python docs
• Real Python

Please let us know if this solves your problem.

Before opening this issue, I tried to read everything I could about python packaging system.
It is quite complicated, but thanks to the PyScaffold, not that much anymore.

I have put print_something() function to src/scaffmodule/__init__.py, so __init__.py now has content:

# -*- coding: utf-8 -*-
from pkg_resources import DistributionNotFound, get_distribution

try:
    # Change here if project is renamed and does not equal the package name
    dist_name = __name__
    __version__ = get_distribution(dist_name).version
except DistributionNotFound:
    __version__ = "unknown"
finally:
    del get_distribution, DistributionNotFound


def print_something():
    print("today is a lovely day")

After installing the sdist package on another computer, I am able to use (I don't know should I say module or package) it as I want:

>>> import scaffmodule
>>> scaffmodule.print_something()
today is a lovely day
>>>

This approach is OK, if using scaffmodule as a library, but what if I want to use it as script - or to make things more complicated - what if I want to use it as standalone shell script, something similar to putup.exe that PyScaffold installs with pip install pyscaffold?

For example, let's say that scaffmodule knows how to send an email. In python code I will import the module, send mail and exit.
Sometimes, it is convenient to send mail from bash script by using argparse parameters.
How to support both options and at the same time use PyScaffold?

Hello @ghuname, I will try to reply to your questions bellow:

Do you want to say that if I delete src/scaffmodule folder and put scaffmodule.py directly into the src folder it will work with no double import (import scaffmodule.scaffmodule)?
Would it be OK, if I move all the code from scaffmodule.py to src/scaffmodule/__init__.py?

This approach will probably work, but I never tested myself. The default configurations generated by PyScaffold are though for a project with more than one file, so it might be the case some minor adjusts are necessary. But in theory I don't see anything preventing it from work.

In this case, will I be able to use command line parameters like: python scaffmodule.py arg1 arg2... ?
Do I need to create a shell script in order to call the module directly (if I don't use it as a library)?
…
This approach is OK, if using scaffmodule as a library, but what if I want to use it as script - or to make things more complicated - what if I want to use it as standalone shell script, something similar to putup.exe that PyScaffold installs with pip install pyscaffold?

There are 2 ways of running scripts from modules/packages installed via pip/PyPI. The first one is to use the -m flag in the python command. For example, you might check that running python -m pyscaffold.cli --help is actually equivalent to putup --help, or that running python -m pip --help is equivalent to pip --help. When you run python -m MODULE (with MODULE following the "import syntax"), Python will actually run the installed module as if it was a script (even __name__ == '__main__' will be True). In the case you run python -m PACKAGE python will try to find a PACKAGE/__main__.py module and run it as a script. This process is described in PEP 338 and __main__.

The second option is to use a setuptools entrypoint. If you have a look on the setup.cfg file generated by PyScaffold you can notice something like the following:

[options.entry_points]
# Add here console scripts like:
# console_scripts =
#     script_name = package.module:function

If you uncomment these lines and change them according to your package structure, when your package is installed, setuptools will create a script_name executable stub in your PATH (or virtual environment's PATH) that will run the function function defined inside package.module every time you run script_name from your console. That is how we do for putup as you can see here and similar to what pip does (the difference is that they use setup.py instead of setup.cfg). Please notice the two approaches are not ortogonal and it is very common to have both working at the same time.

The best way of supporting libraries and executable scripts at the same time in a single Python project is to first define your library API so people importing your project can use it, then create a wrapper function around it that uses argparse (or a similar solution like typer or click) to get the parameters from the command line and call your API function.

For example, if you have a look on the skeleton.py PyScaffold have created for you, you can think about the API being the fib function, so Python scripts can do a from package.skeleton import fib and run fib(42).

Then we have a CLI wrapper around it, defined in the run and main functions. When we register script_name = package.skeleton:run inside setup.cfg > [options.entry_points] > console_script, we will be able to call script_name 42 as if we were calling fib(43) from Python code itself...

Finally, since we also add if __name__ == '__main__': run(), we will also be able to run python -m package.skeleton 42.

Locally I installed scaffmodule package as: python setup.py develop from scaffmodule root folder.
What bothers me is installation on other systems and at the same time ability to use it in its simplest way as import scaffmodule; scaffmodule.print_something().

If you always work in different machines using the pip install workflow, you should be able to uniformly use your package by either importing it from the command line, or by running python -m ... or (if you define an entrypoint) by calling your script name directly.

I hope this helps...

Hi @abravalheri,

I have just created a brand new scaffmodule package. I have created a new src/scaffmodule/demo.py file with contents:

import sys

def print_arguments(*args, **kwargs):
    print(f"{args=}")
    print(f"{kwargs=}")

if __name__ == '__main__':
    print_arguments(*sys.argv[1:])

After making sdist and installing it in new conda environment, I am able to use it with no further actions as you explained before:

(test) C:\Users\user\tmp\scaffmodule\dist>python -m scaffmodule.demo a b c
args=('a', 'b', 'c')
kwargs={}

So, for packages (that contain modules) everything works out of the box beautifully.

Now I added to setup.cfg > [options.entry_points] the following:

console_scripts =
    prn_args = scaffmodule.demo:print_arguments

After reinstalling the package and executing:

(test) C:\Users\user\tmp\scaffmodule\dist>prn_args a b c
args=()
kwargs={}

We can see that we are not collecting sys.argv arguments because we are calling print_arguments() function directly.

According to skeleton.py I modified demo.py in the following way:

import sys

def print_arguments(*args, **kwargs):
    print(f"{args=}")
    print(f"{kwargs=}")

def run_print_arguments():
    print_arguments(*sys.argv[1:])

if __name__ == '__main__':
    run_print_arguments()

and changed setup.cfg to:

console_scripts =
    prn_args = scaffmodule.demo:run_print_arguments

It works:

(test) C:\Users\user\tmp\scaffmodule\dist>prn_args a b c
args=('a', 'b', 'c')
kwargs={}

At this point I can confirm that when PyScaffold is used for a package that contains module(s), everything works out of the box smoothly, which is remarkable.
For such package, everything is set as it should.

Now my dilemma from the very beginning.
As I really like PyScaffold, I would like to use it for a small projects that contain a single module.
I would like to preserve entry_points mechanism and at the same time simplify imports (in order to avoid import scaffmodule.scaffmodule)

@abravalheri Can you please tell me, what should I do now with scaffmodule in order to simplify imports? I can provide zip with scaffmodule if it will help.

@FlorianWilhelm I have just put all the code from demo.py to the init.py, and in setup.cfg changed entry_points to:

console_scripts =
    prn_args = scaffmodule:run_print_arguments

After that, I am able to:

(test) C:\Users\user\tmp\scaffmodule\dist>prn_args a b c d
args=('a', 'b', 'c', 'd')
kwargs={}


(test) C:\Users\user\tmp\scaffmodule\dist>python
Python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scaffmodule
>>> scaffmodule.print_arguments()
args=()
kwargs={}
>>>

Everything works as expected.
Thank you all for all information you have provided.
Now I am able to use PyScaffold for everything. 👍

Hi @ghuname, I also did a quick test here and replacing src/scaffmodule/* with a single file src/scaffmodule.py should also work as expected.

@ghuname

I have just created a new project by using putup scaffmodule on my windows 10 machine. I opened the project in pycharm, added file /scaffmodule/src/scaffmodule/scaffmodule.py with content:

Just an aside, if your module is a single file scaffmodule.py then it normally would not be in a directory. If your module requires multiple files scaffmodule/ then it would have an __init__.py file that represents the module and any other files necessary to implement the module.

Similarly, project directories don't have to be named the same as the module.

Consider this directory structure as an example

.../projects/myproject/src/module1.py
                          /module2/__init__.py
                                  /stuff.py
                                  /morestuff.py

(I hope my indentation survives so the above is readable.)

And then when I package it up and upload it to pypi I name the package "esters_folly".

Enjoy the choices you have with Python. :-)