Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VIA Configurator Refactor #7268

Merged
merged 14 commits into from
Jan 3, 2020
Merged

VIA Configurator Refactor #7268

merged 14 commits into from
Jan 3, 2020

Conversation

wilba
Copy link
Contributor

@wilba wilba commented Nov 5, 2019

The VIA Configurator developers have been busy adding essential features and refining how VIA works internally. We are very close to resuming addition of new keyboards to VIA Configurator, now that we have a much better way of easily adding new keyboards, and with this PR, the required changes to a keyboard's QMK code is minimal and isolated.

This PR adds VIA support through a core feature. The existing code has been moved up into core, and improved in various ways, and all keyboards which currently have VIA support have had their code updated and simplified.

Description

VIA Configurator works through using raw HID communications and dynamic keymaps stored in EEPROM. I put raw HID and dynamic keymaps in QMK core already as these were easily separated from the original code I had originally written for Zeal60. This PR continues that work, and generalizes the required code to enable VIA support while still allowing the keyboard level code to override default behaviour.

As such, for most keyboards that have straightforward requirements, adding VIA support to a keyboard in QMK should be as simple as adding a via keymap directory, containing a default keymap in keymap.c, a rules.mk with VIA_ENABLE = yes, and ensuring the keyboard directory's config.h has unique VENDOR_ID and PRODUCT_ID.

Defining the keyboard for VIA Configurator (the client) has been refactored also. The old method of combining embedded KLE JSON and the LAYOUT_all() macro, has been replaced. Now a single JSON file will define the physical layout, any layout options, and the mapping of physical layout keys to switch matrix positions. The layouts, layout options and switch matrix mapping will be contained in KLE JSON (labels on the key objects), so it is easy to visualize and edit. The keyboard's JSON file can be submitted to a github repo for inclusion into VIA Configurator.

Here is the current guide which explains how to enable VIA Configurator support:

https://github.com/olivia/via-config/wiki/Enabling-VIA-Configurator-Support-in-QMK

Still TODO

  • refactor quantum_keycodes
  • better handling of keyboard-defined and user-defined custom keycodes
  • adding final usage guide to QMK documentation repo

Types of Changes

  • Core
  • Bugfix
  • New feature
  • Enhancement/optimization
  • Keyboard (addition or update)
  • Keymap/layout/userspace (addition or update)
  • Documentation

Copy link
Contributor

@awkannan awkannan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is some of the cleanest code I’ve personally seen in QMK (way better than my own). I will test these tonight and approve once that’s verified, but it looks pretty good to me so far.

quantum/via.c Show resolved Hide resolved
quantum/via.h Show resolved Hide resolved
quantum/via.h Outdated Show resolved Hide resolved
quantum/via.c Outdated Show resolved Hide resolved
quantum/via.h Show resolved Hide resolved
tmk_core/common/keyboard.c Outdated Show resolved Hide resolved
tmk_core/common/keyboard.c Outdated Show resolved Hide resolved
quantum/via.c Outdated Show resolved Hide resolved
quantum/via.c Outdated Show resolved Hide resolved
common_features.mk Outdated Show resolved Hide resolved
common_features.mk Outdated Show resolved Hide resolved
common_features.mk Outdated Show resolved Hide resolved
@drashna
Copy link
Member

drashna commented Nov 5, 2019

Also, one thing that has bothered me, is the macros in the dynamic keymap. Personally, I feel that it should be broken out and made separate from the keymaps, as it is a substantially different feature than keymaps.

Additionally, not everyone would want to use it, but more to the point, plenty of people want to use it without the rest of it (for instance, with the existing dynamic macros feature).

However, this is probably outside of the scope of this PR.

@drashna drashna added breaking_change Changes that need to wait for a version increment core keyboard needs doc labels Nov 5, 2019
@wilba
Copy link
Contributor Author

wilba commented Nov 13, 2019

@drashna why is this labelled breaking_change?

@drashna
Copy link
Member

drashna commented Nov 14, 2019

Because it's a very big change, touching a LOT of files. And because it changes a number of user keymaps

@awkannan
Copy link
Contributor

Because it's a very big change, touching a LOT of files. And because it changes a number of user keymaps

imo a breaking change is something that breaks core behavior, changes an API, etc.

This is just a big change. It's not really breaking imo.

Does it really change user keymaps? As far as I can tell, this doesn't modify any user keymaps. Anyone using VIA for their keyboard knows that changes can happen - Wilba is being super nice fixing it for everyone instead of just breaking things and making people fix it themselves. VIA isn't officially publicly supported, so it's "implement at your own risk".

In terms of NEED DOC, I also don't think that's necessary. yet Until VIA is public again, I wouldn't want people adding things on a whim, etc. I agree that docs should be present when the feature is "ready".

@fauxpark
Copy link
Member

fauxpark commented Nov 14, 2019

These are the changed keymap files that aren't default or via (or some variation). You could argue a few of them are default-ish, but the others seem to be in personal keymaps:

keyboards/hs60/v2/keymaps/goatmaster/rules.mk
keyboards/hs60/v2/keymaps/iso_andys8/rules.mk
keyboards/hs60/v2/keymaps/stanrc85/rules.mk
keyboards/hs60/v2/keymaps/win_osx_dual/rules.mk
keyboards/keebio/iris/keymaps/osiris/rules.mk
keyboards/maartenwut/plain60/keymaps/RGB/rules.mk
keyboards/wilba_tech/rama_works_m10_b/keymaps/knops/config.h
keyboards/wilba_tech/zeal65/keymaps/split_bs/keymap.c

And yeah, we would prefer to see improvements like this broken up into smaller, more manageable logical chunks, as it makes it far easier and faster to review.

@yanfali
Copy link
Contributor

yanfali commented Nov 14, 2019

Because it's a very big change, touching a LOT of files. And because it changes a number of user keymaps

imo a breaking change is something that breaks core behavior, changes an API, etc.

This is just a big change. It's not really breaking imo.

Upas, if someone dumped a large change at you at work what would your reaction be? We want this as much as you do, but smaller more focused pulled requests are easier to digest and merge. You don't have to agree with the labels, they're mostly there for our workflow.

quantum/quantum.c Outdated Show resolved Hide resolved
Copy link
Member

@drashna drashna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's been on my mind for a while now, but I don't think that setting a hard address at the first unused block is the correct way to go about this.

It means that each time that somebody wants to add a new block to be used by EEPROM, that the "size" has to be changed, as well as the magic number incremented/decincemented to reset the eeprom to prevent collisions, again. I don't think that's a good experience for either said, and it puts more onto us.

I think a better long term solution would be to put via at the end of the eeprom. Find the last byte/block, and calculate back from that. For ATmega32u4, this should be a set value, as the EEPROM size appears to be 1024 bytes. For ARM, this is a bit more tricky, though the info/setting should be in the repo (I think we set it to 4096 bytes by default for chibiOS's faux EEPROM).

I don't think this is something that has to be in this PR, but I think it's something that would be a good change, for the future, as it should help prevent collisions and additional work, once it's completed.

@wilba
Copy link
Contributor Author

wilba commented Nov 24, 2019

@drashna Changing EECONFIG_SIZE whenever QMK Core's usage of EEPROM is increased is not only trivial, it gives all QMK users (not just VIA) the ability for their code to auto adapt to any change. It is no different to how the SAFE_RANGE keycode works.

VIA uses the build date as the magic number/version number stored in EEPROM to detect new firmware and cause EEPROM reset to defaults. No system of magic number is perfect, but this is good enough for "official" releases of firmware, which won't happen on the same date, and does not require manual incrementing/decrementing. Also, bootmagic lite will be the recommended way of updating firmware, which makes EEPROM out-of-sync errors even less frequent.

Technically, VIA is at the end of the EEPROM by default, the dynamic keymaps/macros take up the available EEPROM (on ATmega32u4) and it has been designed this way because EEPROM is limited on Atmega32u4, and it is more logical to allocate in an incremental way after QMK Core's EEPROM usage. Allocating from the end of available EEPROM up to QMK Core's EEPROM usage would still require a limit such as EECONFIG_SIZE, so I don't see any advatage in doing this.

At some point, if QMK Core has an API/configuration for EEPROM usage (esp. disabling QMK Core's usage that is not needed for a keyboard), then we can switch to that.

Note also, ARM users could configure where VIA and dynamic keymaps uses EEPROM by overriding symbols.

Copy link
Member

@nooges nooges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For anyone who wants to review the PR, I split up the first commit into a few to help with digestion: https://github.com/nooges/qmk_firmware/commits/via_refactor_pr_mod

Copy link
Member

@zvecr zvecr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't had time to look through the changes, but the reason this is still labelled breaking change is due to user keymaps being changed. There might be people here that have a definition of what that term means for them, but for QMK right now, one general rule is no user keymap modification without prior knowledge.

If all users can sign off here, then we can skip that process.

@wilba
Copy link
Contributor Author

wilba commented Dec 8, 2019

@zvecr The only changes to "user" keymaps are replacing the redundant rules.mk with one that just enables VIA, which is how the original keymap was configured, even though technically that's a strange combination, wanting your "keymap" stored in the QMK repo but still building VIA enabled firmware. The other changes in keymaps are refactoring the default and via keymaps of PCBs that I maintain or which I put into VIA.

@wilba
Copy link
Contributor Author

wilba commented Dec 22, 2019

Merged master again.

I have updated @Boy-314 rules.mk (#7638) to the new way of enabling VIA.

Regarding "user" changes, I believe it's better to change them so that they continue to enable VIA as before (i.e. compile with exactly the same feature set), than leave them unchanged and effectively "disable" VIA.

Can someone please review that the changes to these rules.mk are confined to adapting to the new way of enabling VIA?

@mechmerlin
Copy link
Contributor

@andys8 @stanrc85 @mcmadhatter @sidcarter

Some large changes coming in and changes have been made to each of your keymaps. Are you guys ok with the changes to your keymaps?

@mcmadhatter
Copy link
Contributor

@mechmerlin changes to hs60/v2/win_osx_dual are fine by me.

Copy link
Contributor

@evyd13 evyd13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great guys! (Changes to Plain60 keyboard)

@stanrc85
Copy link
Contributor

stanrc85 commented Jan 2, 2020

@mechmerlin I don't use VIA so I assume this won't affect anything I have in my QMK user space.

Copy link
Member

@noroadsleft noroadsleft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to review the parts I was likely to understand. The overwhelming majority of the C changes are over my head as I have zero experience with EEPROM or VIA, and little skill in C.

Files I didn't review:

  • keyboards/cannonkeys/an_c/config.h
  • keyboards/cannonkeys/instant60/config.h
  • keyboards/cannonkeys/iron165/config.h
  • keyboards/cannonkeys/satisfaction75/satisfaction75.c
  • keyboards/cannonkeys/satisfaction75/satisfaction_encoder.c
    • At a glance this looks like only function and macro renames; not inclined to investigate further at the moment.
  • keyboards/cannonkeys/savage65/config.h
  • keyboards/cannonkeys/stm32f072/keyboard.c
  • keyboards/cannonkeys/tmov2/config.h
  • keyboards/hs60/v2/config.h
  • keyboards/nk65/config.h
  • keyboards/projectkb/alice/config.h
    • Looks to be using the Cannonkeys STM32F072 RGB Underglow implementation, and not actually VIA.
  • keyboards/wilba_tech/wt_main.c
  • quantum/dynamic_keymap.c
    • Didn't feel like checking all the math in here; might come back to this.
  • quantum/via.c
    • Everything up to Line 101 plus process_record_via() looks okay to me. The rest is a bit (or a lot) over my head.
  • quantum/via.h
    • Too much math for how non-alert I am at the moment.
  • tmk_core/common/eeconfig.h
    • Looks okay on its own, but haven't investigated how it's used by the rest of the changes.

keyboards/wilba_tech/zeal60/keymaps/via/keymap.c Outdated Show resolved Hide resolved
keyboards/wilba_tech/zeal65/zeal65.h Show resolved Hide resolved
quantum/via.c Outdated Show resolved Hide resolved
Copy link
Member

@noroadsleft noroadsleft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K, came back and took another run at this.

@awkannan helped me understand some of the bits I didn't understand the first time around. The only changes I'm not following at this point are to:

  • keyboards/cannonkeys/satisfaction75/satisfaction75.c
  • keyboards/wilba_tech/wt_main.c

... and that's just because that level of C is above my skillset. I understand what it's doing conceptually, but not practically, and that's only fixed if I actually learn C. So those two files are __attribute__ ((weak)) ✔️

If the remaining affected users sign off on the keymap changes (or if they don't do so promptly), I'm comfortable enough with this getting merged.

@wilba
Copy link
Contributor Author

wilba commented Jan 3, 2020

@noroadsleft Originally, wt_main.c contained the specifics of VIA protocol, i.e. actually handling messages and routing them to dynamic keymaps functions. My keyboard PCBs (and some others) would include that source file. When @awkannan made Satisfaction 75, he copied that code into satisfaction75.c and extended the protocol for his keyboard specifics (encoder handling, etc.)

This PR refactors the common code that was in wt_main.c and satisfaction75.c into via.c, leaving in those files the specific handling for those keyboards. This is much like how QMK works in other areas, how something like matrix handling can be overridden or extended at the keyboard level with things like strong linked matrix_scan_kb() functions. Thus, via.c handles everything for keyboards that just need the dynamic keymaps and macros handling, and keyboards like Satisfaction 75 can extend upon that by handling only the specific protocol (i.e. what's different) and leaving the common handling to via.c in QMK Core.

@awkannan
Copy link
Contributor

awkannan commented Jan 3, 2020

@sidcarter @andys8 Pinging you guys again as we need your signoff for this merge, since your user keymaps were impacted. Are you guys ok with the changes to your keymaps?

@noroadsleft
Copy link
Member

@Wilba6582 That's pretty much exactly what I thought it was doing. I did compare the parts I didn't understand from each file against each other and saw the minor differences. @awkannan's reminder about VIA working differently on his boards due to hardware changes (OLED, encoder, lighting implementations) made me realize it was all probably stuff that was specific to his PCBs.

Great explanation, and much appreciated. 👍

@andys8
Copy link
Contributor

andys8 commented Jan 3, 2020

I'm okay with the changes being made and thankful for your contribution. I've to add I can't currently test the changes and I can't confirm there is no breakage.

@sidcarter
Copy link
Contributor

sidcarter commented Jan 3, 2020

@olivia @mechmerlin I'm good with the changes to my keymap 👍

Edit: @awkannan

Also, looking forward to this merge very muchly :D

@Boy-314
Copy link
Contributor

Boy-314 commented Jan 3, 2020

I approve

@noroadsleft noroadsleft removed awaiting review breaking_change Changes that need to wait for a version increment labels Jan 3, 2020
@drashna drashna merged commit 320822d into qmk:master Jan 3, 2020
@drashna
Copy link
Member

drashna commented Jan 3, 2020

Thanks!

HokieGeek pushed a commit to HokieGeek/qmk_firmware that referenced this pull request Feb 21, 2020
* VIA Refactor

* Remove old code

* review changes

* review changes

* Fix cannonkeys/satisfaction75/prototype:via build

* Add via.h to quantum.h

* Move backlight init to after backlight config load

* Merge branch 'master' into via_refactor_pr

* Update user's rules.mk to new way of enabling VIA

* Added id_switch_matrix_state

* Review changes
kylekuj pushed a commit to kylekuj/qmk_firmware that referenced this pull request Apr 21, 2020
* VIA Refactor

* Remove old code

* review changes

* review changes

* Fix cannonkeys/satisfaction75/prototype:via build

* Add via.h to quantum.h

* Move backlight init to after backlight config load

* Merge branch 'master' into via_refactor_pr

* Update user's rules.mk to new way of enabling VIA

* Added id_switch_matrix_state

* Review changes
@wilba wilba deleted the via_refactor_pr branch March 23, 2021 23:11
BorisTestov pushed a commit to BorisTestov/qmk_firmware that referenced this pull request May 23, 2024
* VIA Refactor

* Remove old code

* review changes

* review changes

* Fix cannonkeys/satisfaction75/prototype:via build

* Add via.h to quantum.h

* Move backlight init to after backlight config load

* Merge branch 'master' into via_refactor_pr

* Update user's rules.mk to new way of enabling VIA

* Added id_switch_matrix_state

* Review changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.