Skip to content

Commit

Permalink
re-organize framework code in to framework/, user code in to app/
Browse files Browse the repository at this point in the history
  • Loading branch information
floodyberry committed Aug 12, 2014
1 parent 5ac6c91 commit 1443cdb
Show file tree
Hide file tree
Showing 58 changed files with 104 additions and 64 deletions.
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ bin/*
build/*
build_util/*
config.log
src/driver/asmopt.h
src/driver/asmopt_internal.h
src/util_implementations.h
framework/include/asmopt.h
framework/include/asmopt_internal.h
framework/include/util_implementations.h
example
example-util
!example/
Expand Down
24 changes: 12 additions & 12 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ BASEDIR = .
BINDIR = bin
BUILDDIR = build
BUILDDIRUTIL = build_util
INCLUDE = $(addprefix -I$(BASEDIR)/,src src/extensions src/driver $(addsuffix $(ARCH)/,src/driver/) include)
INCLUDE = $(addprefix -I$(BASEDIR)/,app/extensions app/include framework/include framework/driver framework/driver/$(ARCH))
CINCLUDE = $(INCLUDE)
ASMINCLUDE = $(INCLUDE)

Expand All @@ -27,22 +27,22 @@ endif
#
rwildcard = $(foreach d, $(wildcard $(1)*), $(call rwildcard, $(d)/, $(2)) $(filter $(subst *, %, $(2)), $(d)))

SRCDRIVER = $(call rwildcard, src/driver/, *.c)
SRCEXT = $(call rwildcard, src/extensions/, *.c)
SRCDRIVER = $(wildcard framework/driver/*.c)
SRCEXT = $(call rwildcard, app/extensions/, *.c)
SRCASM =
SRCMAIN = src/main.c
SRCUTIL = src/util.c
SRCSHARED = src/shared.c
SRCUTIL += $(call rwildcard, src/util/, *.c)
SRCMAIN = app/main.c
SRCUTIL = framework/main_util.c framework/bench.c framework/fuzz.c
SRCSHARED = framework/main_shared.c


# do we have an assembler?
ifeq ($(HAVEAS),yes)

# grab all the assembler files
SRCASM = $(call rwildcard, src/extensions/, *.S)
SRCASM = $(call rwildcard, app/extensions/, *.S)

# add asm for the appropriate arch
SRCASM += $(call rwildcard, $(addsuffix $(ARCH),src/driver/), *.S)
SRCASM += $(call rwildcard, $(addsuffix $(ARCH),framework/driver/), *.S)

endif

Expand Down Expand Up @@ -90,7 +90,7 @@ exe: makebin $(BINDIR)/$(PROJECTNAME)$(EXE)
install-generic:
$(INSTALL) -d $(includedir)/lib$(PROJECTNAME)
$(INSTALL) -d $(libdir)
$(INSTALL) -m 644 include/*.h $(includedir)/lib$(PROJECTNAME)
$(INSTALL) -m 644 app/include/*.h $(includedir)/lib$(PROJECTNAME)

lib: makebin $(BINDIR)/$(PROJECTNAME)$(STATICLIB)
@echo built [$(BINDIR)/$(PROJECTNAME)$(STATICLIB)]
Expand All @@ -112,8 +112,8 @@ ifneq ($(SOIMPORT),)
$(INSTALL) -m 755 $(BINDIR)/$(SONAME) $(bindir)
$(INSTALL) -m 644 $(BINDIR)/$(SOIMPORT) $(libdir)
else ifneq ($(SONAME),)
ln -f -s $(SONAME) $(libdir)/lib$(PROJECTNAME).$(SOSUFFIX)
$(INSTALL) -m 755 $(SONAME) $(libdir)
$(INSTALL) -m 755 $(BINDIR)/$(SONAME) $(libdir)
ln -f -s $(libdir)/$(SONAME) $(libdir)/lib$(PROJECTNAME).$(SOSUFFIX)
endif
else
shared:
Expand Down
21 changes: 11 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,18 @@ x86 is fully supported, and the first pass at ARM support is now in!
# QUICK OVERVIEW #

* Write once, run everywhere assembler, using GCC and Yasm.
* Project name is set in [project.def](project.def) and version is set in [project.ver](project.ver)
* Platform specific code (cpu feature detection, cpu cycles, assembler macros) is in `src/driver/platform`
* Optimized implementations go in `extensions/name` and are exposed through `include/name.h`
* Sample `main.c` and fuzzing / benchmarking support is in [src](src).
* Project name is set in [app/project.def](app/project.def) and version is set in [app/project.ver](app/project.ver)
* Platform specific code (cpu feature detection, cpu cycles, assembler macros) is in `framework/driver/platform`
* Optimized implementations go in `app/extensions/name` and are exposed through `app/include/name.h`
* Fuzzing / benchmarking support is in [framework/](framework/).
* Sample [app/main.c](app/main.c) provided showing use of cpuid and calling the example extension.
* Platforms supported are x86, ARM.

# HOW IT WORKS #

I really wanted to avoid this, but before anything is done, a configuration script must be run to determine compiler capabilities, instruction sets supported, and so on. This is both for the assembler to know what it can assemble, and for the C code which will use the assembler to know which versions it can use.

gcc and Yasm each have their own bootstrap file ([gcc\_driver.inc](driver/gcc_driver.inc) and [yasm\_driver.inc](driver/yasm_driver.inc)) which handles determining platform, compiler, and setting up the macros needed. The initial file that includes the bootstrap code must have an `.S` extension to allow the C preprocessor to run for gcc compatible implementations. The C preprocessor macros are however only used to set up the GNU assembler macros which will be used by the assembler files; this is because there is no way to include a file from a macro with the C preprocessor.
gcc and Yasm each have their own bootstrap file ([gcc\_driver.inc](framework/driver/gcc_driver.inc) and [yasm\_driver.inc](framework/driver/yasm_driver.inc)) which handles determining platform, compiler, and setting up the macros needed. The initial file that includes the bootstrap code must have an `.S` extension to allow the C preprocessor to run for gcc compatible implementations. The C preprocessor macros are however only used to set up the GNU assembler macros which will be used by the assembler files; this is because there is no way to include a file from a macro with the C preprocessor.

## BOOTSTRAPPING ##

Expand Down Expand Up @@ -112,9 +113,9 @@ For the moment, non-x86 platforms are gcc only, so you may use standard #if defi

## CPUID ##

CPUID implementations are in `driver/arch/` and exposed through [cpuid.c](driver/cpuid.c) with `unsigned long cpuid(void)`.
CPUID implementations are in `framework/driver/arch/` and exposed through [cpuid.c](framework/driver/cpuid.c) with `unsigned long cpuid(void)`.

The [x86 cpuid](driver/x86/cpuid_x86.S) detects everything from MMX up to (theoretically, based on Intel's programming reference) AVX-512. The implementation "cheats" by having the bootstrap provide `CPUID_PROLOGUE` and `CPUID_EPILOGUE` so a single implementation can be used for both x86 and x86-64.
The [x86 cpuid](framework/driver/x86/cpuid_x86.S) detects everything from MMX up to (theoretically, based on Intel's programming reference) AVX-512. The implementation "cheats" by having the bootstrap provide `CPUID_PROLOGUE` and `CPUID_EPILOGUE` so a single implementation can be used for both x86 and x86-64.

Also provided are example runtime dispatching functions to test and select the optimal version based on the current CPU.

Expand Down Expand Up @@ -207,9 +208,9 @@ If you are getting `/usr/bin/ld: error: cannot find -lexample` when trying to li

### NAME ###

The name of the project is set in [project.def](project.def). This is used to create project specific function names using `LOCAL_PREFIX`.
The name of the project is set in [app/project.def](app/project.def). This is used to create project specific function names using `LOCAL_PREFIX`.

The project version is in [project.ver](project.ver). Unused except for shared library names on some *nix's.
The project version is in [app/project.ver](app/project.ver). Unused at the moment except for shared library names on some *nix's.

### CONFIGURING ###

Expand Down Expand Up @@ -266,7 +267,7 @@ Well some may not be used yet, but you know, for the future.

I've got the Visual Studio project generator working! It generates a Visual Studio [2010,2012,2013] solution with projects for a static library, dynamic library, and utility project for both 32 and 64 bits. Generated files (exe, lib, dll) are currently placed in `asm-opt/bin/[Release|Debug]/[amd64|x86-32bit]/`.

It only requires that [yasm.exe](http://yasm.tortall.net/Download.html) be somewhere in the system path for Visual Studio to execute. You can place `yasm.exe` in the `vs2010/` directory if you're especially lazy.
It only requires that [yasm.exe](http://yasm.tortall.net/Download.html) be somewhere in the system path for Visual Studio to execute. You can place `yasm.exe` in the visual studio directory if you're especially lazy.

php genvs.php [options]

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@ example(const unsigned char *arr, size_t count) {

#include <stdio.h>
#include <string.h>
#include "util/fuzz.h"
#include "util/bench.h"
#include "fuzz.h"
#include "bench.h"

static const fuzz_variable_t fuzz_inputs[] = {
{"input", FUZZ_RANDOM_LENGTH_ARRAY0, 16384},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,8 @@ secure_compare32_bootup(const unsigned char *x, const unsigned char *y) {

#if defined(UTILITIES)

#include "util/bench.h"
#include "util/fuzz.h"
#include "bench.h"
#include "fuzz.h"

static const fuzz_variable_t fuzz_inputs[] = {
{"input32x", FUZZ_ARRAY, 32},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ secure_zero_bootup(unsigned char *p, size_t len) {

#if defined(UTILITIES)

#include "util/bench.h"
#include "util/fuzz.h"
#include "bench.h"
#include "fuzz.h"

static const fuzz_variable_t fuzz_inputs[] = {
{"input", FUZZ_RANDOM_LENGTH_ARRAY0, 16384},
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
34 changes: 25 additions & 9 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,13 @@ test_for_gnu_make() {
}

# options
outputheader="src/driver/asmopt.h"
outputheaderstub="src/driver/asmopt_internal.h"
outpututilimpls="src/util_implementations.h"
outputheader="framework/include/asmopt.h"
outputheaderstub="framework/include/asmopt_internal.h"
outpututilimpls="framework/include/util_implementations.h"
outputmake="asmopt.mak"
outputlog="config.log"
inputprojectname="app/project.def"
inputprojectver="app/project.ver"
asmheader=".text"
debug="no"
bits32="no"
Expand Down Expand Up @@ -261,12 +263,16 @@ ARCH="generic"
OFORMAT="generic"
EXE=""
STATICLIB=".lib"
PROJECTNAME=`tr -d '\n' <project.def | tr '[:upper:]' '[:lower:]'`
PROJECTNAMEUPPER=`tr -d '\n' <project.def | tr '[:lower:]' '[:upper:]'`
PROJECTVERMAJOR=`cut -d . -f 1 project.ver`
PROJECTNAME=`tr -d '\n' <$inputprojectname | tr '[:upper:]' '[:lower:]'`
PROJECTNAMEUPPER=`tr -d '\n' <$inputprojectname | tr '[:lower:]' '[:upper:]'`
PROJECTVERMAJOR=`cut -d . -f 1 $inputprojectver`

if [ x"$PROJECTNAME" = "x" ]; then
fatal_error "Unable to determine project name! Edit project.def to set it"
fatal_error "Unable to determine project name! Edit $inputprojectname to set it"
fi

if [ x"$PROJECTVERMAJOR" = "x" ]; then
fatal_error "Unable to determine project name! Edit $inputprojectver to set it"
fi

if [ "$yasm" = "yes" ]; then
Expand Down Expand Up @@ -576,6 +582,16 @@ else
SOFLAGS="-shared -Wl,-soname,\$(SONAME) $SOFLAGS"
fi

if [ `uname` = "SunOS" ]; then
if test -x /usr/ucb/install; then
INSTALL=/usr/ucb/install
elif test -x /usr/bin/ginstall; then
INSTALL=/usr/bin/ginstall
elif test -x /usr/gnu/bin/install ; then
INSTALL=/usr/gnu/bin/install
fi
fi

CFLAGS="$CFLAGS -DBUILDING_ASMOPT_${PROJECTNAMEUPPER}"

cat > $outputmake << EOF
Expand Down Expand Up @@ -652,8 +668,8 @@ EOF

# build util_implementations.h

util_includes=`ls include/*.h | sed "s/include\/\(.*\)\.h/#include \"\1.h\"/"`
util_impls=`ls include/*.h | sed "s/include\/\(.*\)\.h/\tmake_impl(\1),/"`
util_includes=`ls app/include/*.h | sed "s/app\/include\/\(.*\)\.h/#include \"\1.h\"/"`
util_impls=`ls app/include/*.h | sed "s/app\/include\/\(.*\)\.h/make_impl(\1),/"`

cat > $outpututilimpls << EOF
$util_includes
Expand Down
8 changes: 4 additions & 4 deletions docs/EXAMPLE.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
An [example](extensions/example/) is provided demonstrating how to implement, select, and call an optimized function. The example algorithm being implemented is summing up the ~~~little-endian~~~ 32-bit signed integers in a given array and returning the result.
An [example](app/extensions/example/) is provided demonstrating how to implement, select, and call an optimized function. The example algorithm being implemented is summing up the ~~~little-endian~~~ 32-bit signed integers in a given array and returning the result.

### PUTTING IT TOGETHER ###

#### ASSEMBLER ####

[example.S](extensions/example/example.S) includes the bootstrap header and aggregates all the implementations in to a single file:
[example.S](app/extensions/example/example.S) includes the bootstrap header and aggregates all the implementations in to a single file:

#if defined(__GNUC__)
#include "gcc_driver.inc"
Expand All @@ -20,7 +20,7 @@ An [example](extensions/example/) is provided demonstrating how to implement, se
INCLUDE_IF_SSE2_32BIT "example/example_sse2-32.inc"
INCLUDE_IF_X86_32BIT "example/example_x86-32.inc"

with an assembler implementation, [example_x86-32.inc](extensions/example/example_x86-32.inc), looking (something like) like:
with an assembler implementation, [example_x86-32.inc](app/extensions/example/example_x86-32.inc), looking (something like) like:

SECTION_TEXT

Expand All @@ -46,7 +46,7 @@ Suggested function naming is name_extension, e.g. `example_x86`, `aes_avx2`, `cu

#### C ####

The C code "glue" is [impl.c](extensions/example/impl.c), which starts by declaring an implementation struct holding the cpu flags for the implementation and its methods. Note that an optimized implementation may have multiple methods, such as a signing function which would have separate functions for key generation, signing, and verifying:
The C code "glue" is [impl.c](app/extensions/example/impl.c), which starts by declaring an implementation struct holding the cpu flags for the implementation and its methods. Note that an optimized implementation may have multiple methods, such as a signing function which would have separate functions for key generation, signing, and verifying:

typedef struct example_impl_t {
uint32_t cpu_flags;
Expand Down
4 changes: 2 additions & 2 deletions docs/UTILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Extension agnostic utilities are provided for

## FUZZING ##

Abstract fuzzing is provided through [fuzz.h](src/util/fuzz.h) and [fuzz.c](src/util/fuzz.c). Extending an example to support fuzzing requires 1 simple function to be passed to the fuzzer, which then handles the work of selecting available implementations, producing random numbers, producing the input, collecting the output, comparing the output, detecting and displaying mismatches, and displaying the progress using a simple counter so you can tell progress is being made.
Abstract fuzzing is provided through [fuzz.h](framework/include/fuzz.h) and [fuzz.c](framework/fuzz.c). Extending an implementation to support fuzzing requires 1 simple function to be passed to the fuzzer, which then handles the work of selecting available implementations, producing random numbers, producing the input, collecting the output, comparing the output, detecting and displaying mismatches, and displaying the progress using a simple counter so you can tell progress is being made.

### EXAMPLE ###

Expand Down Expand Up @@ -38,7 +38,7 @@ An extension must tell the fuzzer what inputs it will need, and the output it wi

* `FUZZ_DONE` indicates the end of the list, `size` is ignored
* `FUZZ_ARRAY` specifies an array of `size` 8 bit unsigned integers
* `FUZZ_RANDOM_LENGTH_ARRAY[0-3]` specifies an of [0-`size`) 8 bit unsigned integers, where the size is determined randomly at run-time. This _may_ be specified for output as well as input, and the size in the output must match the input size. How to determine the sizes in the next section.
* `FUZZ_RANDOM_LENGTH_ARRAY[0-3]` specifies an array of [0-`size`) 8 bit unsigned integers, where the size is determined randomly at run-time. This can be specified for output as well as input, and the size in the output must match the input size. How to determine the sizes in the next section.

The declaration for our example extension is:

Expand Down
2 changes: 1 addition & 1 deletion src/util/bench.c → framework/bench.c
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#include <time.h>
#include "cpucycles.h"
#include "cpuid.h"
#include "util/bench.h"
#include "bench.h"

/* a 32k, 64 byte aligned buffer to bench with */
unsigned char *
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion src/util/fuzz.c → framework/fuzz.c
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#include <string.h>
#include <time.h>
#include "cpuid.h"
#include "util/fuzz.h"
#include "fuzz.h"

/*
Chacha/8 rng with no addition of state words post-mixing, no security at all, but good
Expand Down
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion src/driver/cpuid.h → framework/include/cpuid.h
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ enum cpuid_flags_generic_t {
CPUID_GENERIC = (0)
};

#include "cpuid_impl.h"
#include "cpuid_flags.inc"

unsigned long LOCAL_PREFIX(cpuid)(void);

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 1443cdb

Please sign in to comment.