[00:21:30] jmvalin joins the room [01:04:10] jmvalin leaves the room [01:05:11] jmvalin joins the room [01:34:52] jmspeex leaves the room [02:05:30] jmvalin leaves the room [02:08:01] jmvalin joins the room [06:35:22] jmspeex joins the room [07:23:25] jmspeex leaves the room [08:10:18] jmspeex joins the room [08:33:52] burn joins the room [08:34:31] Kevin, did you mean for the subject links to be for IETF 77? [08:45:56] jmspeex leaves the room [09:01:48] jmspeex joins the room [09:22:26] jmspeex leaves the room [10:36:22] mlm.michael.miller joins the room [10:39:30] mlm.michael.miller leaves the room [11:02:36] JonathanLennox joins the room [11:03:31] jmspeex joins the room [11:06:55] YJS joins the room [11:07:11] stephanwenger joins the room [11:07:27] jmspeex leaves the room [11:08:01] jmspeex joins the room [11:08:26] as0-d91k joins the room [11:08:32] christopher.blizzard joins the room [11:08:55] Minute-Taker joins the room [11:09:43] Jonathan: Note Well (shows slides on Note Well and IPR Policy) [11:09:45] Barry Leiba joins the room [11:10:03] Jonathan: A month is timely, 3 months is not, after RFC is issued, it is not timely at all! [11:10:18] sorenskakjensen joins the room [11:10:22] resnick joins the room [11:10:27] jmspeex leaves the room [11:10:31] csp joins the room [11:10:43] RjS joins the room [11:10:48] jr: ipr disclosure: 1 month is reasonable, before last call is maybe reasonable, rfc publication is not [11:11:12] agenda (someone have the web page with the documents? Paste in please) [11:11:18] bashes to the agenda? [11:11:19] Joanthan: Administrivia and Agenda (for delight and amusement). WG status update, followed by codec development update. [11:11:28] https://datatracker.ietf.org/meeting/78/materials.html [11:11:40] tomkri joins the room [11:11:43] Hm. We don't need two people typing into jabber. [11:11:59] I thought BA was for minutes and I was for jabber scribing? [11:12:06] That's what I thought too. [11:12:19] Joanthan: WG dashboard. Codec standardization guidelines: Red! (was to go to WGLC in April, not done. Codec Rqeuirements: Yellow. ETA was June 2010. [11:12:29] Codec spec: Green. ETA: was WGC July 2011. [11:12:42] Bernard: Are you jabber scribing? [11:14:09] Pete: I am just taking notes... someone else volunteered to handle comments. [11:14:22] Ah. [11:14:30] OK. I'll hang back. [11:14:41] Liaison status: ITU SG-16 communicated in May 2010, a resonse was sent in mid-May. [11:14:57] suzukisn joins the room [11:15:01] 3GPP: response received. [11:16:02] Prototype for IETF Interactive Audio Codec: draft-valin-codec-prototype-01.txt [11:16:25] Cullen Jennings joins the room [11:16:43] Working on prototype to mix functionality of SILK and CELT... not much overlap in functionality.... believe it meets quite a few of the requirements. [11:16:47] csp leaves the room [11:16:59] want feedback... are there missing areas? [11:17:11] tonyhansen joins the room [11:17:36] IETF-78 codec agenda Slides Codec Prototype Rate Adaptation Audio Codec Quality Tests Codec Requirements Discussion Codec Guidelines Discussion Codec Working Group Status Session 2010-07-26 1300-1500: Auditorium 2 - Audio stream - codec chatroom iCal: ietf-78-codec.ics Agenda Codec WG agenda - IETF 78 (Maastricht), Monday, July 26, from 13:00-15:00 in Auditorium 2 ====================================================================== 1. Administrivia, including overall WG status update (10 min) 2. Codec development and collaboration progress to date (30 min, ref: draft-valin-codec-prototype) 3. Technical requirements update (review and closure of open action items) (20 min, ref: draft-ietf-codec-requirements-00) 4. Rate adaptation (10 min) 5. Codec testing methodology background and test house / workflow proposal (45 min incl open mic discussion, ref: draft-valin-codec-guidelines) 6. Summary / action items (5 min) [11:18:53] Work Since Last Meeting: SILK pre-integration work... need SILK and CELT to use same range coder....are using CELT coder. [11:19:18] Fabricio joins the room [11:19:36] CELT pre-integration work: optimization of 20 ms frames... adjustable frequence range, so some freq could be handled by SILK, others by CELT. [11:20:02] Final step was integration work during the last few weeks.... [11:20:23] Where is the code: publicly accessible with git: git://git.xiph.org/celt.git [11:20:23] tomkri leaves the room [11:20:48] (For CELT). Temp SILK repository: git://git.xiph.org/users/jm/silk.git [11:21:01] Jean-Marc's version of Hybrid has a git repository as well. [11:21:42] Hybrid Characteristics: Sampling rates: 8 - 48 kHz, audio bandwidth 4-20 KHz [11:22:18] Bitrates: ~6-28 kb/s [11:23:45] Overview: signal fed to both CELT (8+ kHz ) and SILK (0 - 8 kHz) [11:24:08] tomkri joins the room [11:24:29] burn leaves the room [11:24:35] 32 operating modes. [11:24:50] SILK-only, SILK + CELT, CELT-only. [11:26:09] Listening Test Results: 5 of 8 were Skype employees.... [11:26:24] Not a serious measurements, but more of an example of how evaluation could be done. [11:27:10] burn joins the room [11:27:42] resnick leaves the room [11:28:13] resnick joins the room [11:28:15] 8 different modes, 3 were reference. Orignla Full-bank, second is hybrid (close to original). [11:28:19] resnick leaves the room [11:28:45] Conclusion: could be a prototype codec for speech and music... designed to match draft-ietf-codec-requirements... future integration work [11:29:11] mode selection/switching, better hybride mode bit allocation, closer PLC integration, lots of tuning and more code sharing needed. [11:29:28] tomkri leaves the room: Replaced by new connection [11:29:31] tomkri joins the room [11:29:54] Jonathan: will get some samples to play out..... [11:30:07] dwd joins the room [11:30:15] Plays original at 48 khz... [11:30:31] tomkri has set the subject to: Codec WG meeting IETF-78 | http://tools.ietf.org/wg/codec/agenda?item=agenda78.html [11:30:40] Next will play hybrid at 32 kB/s [11:30:56] will try the 24.... [11:31:17] (Of course, listening to these remotely, they all sound the same...) [11:31:33] juberti joins the room [11:31:51] resnick joins the room [11:32:11] heh [11:32:32] stephanwenger leaves the room [11:32:37] Jonathan: for giggles... clear progress.... tests are somewhere between impressive and fabulous... we will take a hum... interest in adopting document as a WG work item? [11:32:54] Joe Hildebrand joins the room [11:33:07] Stephan Wenger: I want to look at the document before humming. [11:33:18] Jonathan: feel free not to hum... there is a document. [11:33:38] Jonathan: All in favor, please hum..... All opposed? hum... [11:33:50] Cullen Jennings: fairly strong consensus in favor of adoption. [11:34:18] Jonathan: let the notes show that there was consensus in favor. (Noted). [11:34:37] Michael Knappe: Codec requirements update [11:35:13] Current version : draft-ietf-codec-requirements-00, Jean-Marc Valin and Koen Vos are editors. [11:35:28] Action from March was to explore and clarify a number of design chocies. [11:35:51] Discussion points: how are sample rates selected, and which ones? [11:36:20] Proposal: treat within SDP as separate codecs. [11:36:54] Jean-Marc: sample rates don't need to be treated as separate in SDP... in-band you can change rates. [11:37:31] Jean-Marc: Each side can sample at different rates and can understand each other... so need to put in SDP..... [11:38:24] Jean-Marc: idea (not entirely implemented) is for audio bandwidth to automatically scale with sampling rate... at SDP level may be worth signaling preference for audio bandwidth, but treated as a single codec for compatibility reasons. [11:38:40] Jean-Marc: an AVT issue, more than a CODEC issue. [11:39:09] Cullen: as we move forward, it might become more clear.... also it might not be a topic for this WG, as opposed to MMUSIC... even thought it might be the same people. [11:39:24] Cullen: If you have thoughts (including taking it out of the doc), please speak up. [11:40:06] Jonathan Lennox: SDP can describe bandwidth... but no need for a whole lot beyond that... an advantage to minimizing number of possibilities. [11:40:22] Jonathan Lennox: need to negotiate clock rate, but otherwise.... [11:40:42] Stephan Wenger: sample rate is input sample rate of the codec? Or the audio bw? [11:41:08] Stephan Wenger: internal sampling rate.... I would signal in SDP the highest sample rate supported.... [11:41:28] Stephan Wenger: include all lower sample rates... and single the selected one in-band. [11:41:56] Colin Perkins: distinguish between RTP clock rate sampled in SDP (highest rate), keep that separate from audio sampling and how codec is working. [11:42:22] Colin Perkins: Process: would make sense for a codec format to be produced and discussed in AVT. [11:42:37] Michael: Should also take multi-channel issues to AVT. [11:43:52] Jonathan: are trying to close issues on reqts. doc. [11:44:36] Christian, University of Tuebengen: might want to change 24 to 12 (if internal sampling frequencies). Problem with maximal coding rate... [11:45:06] Christian: could be problems supporting upper rate, while retaining flexibility to switch between different nodes. [11:46:00] (Someone): would not support idea of signalling separate codecs... different from layering and how that works. [11:46:18] Michael: we should come up with a very specific proposal and publish on the mailing list.... [11:46:56] Michael: desire for bit exact fixed point/lower complexity mode.... [11:48:06] Jean-Marc: current reqts. draft says bit-exactness is not a requirement (encoder or decoder)... just "close to reference".... [11:48:32] Jean-Marc: do people agree with what it currently in the doc? [11:49:28] Christian: don't understand need for bit exactness fully... should we discuss on the list? [11:49:58] Jean-Marc: is useful if people can use fixed-point and be bit-exact... but at same time can allow others to not be bit-exact if they spend time to verify that output is correct. [11:50:44] (Someone): reqt comes from telco desiring guaranteed quality... but not a strong requirement for IETF.... [11:51:34] (Someone) second motivation comes from certain lawyers, to clearly specify conformance due to licensing mechanisms (which require conformance as a pre-requisite). That also is not super critical in this WG. [11:51:48] Joe Hildebrand leaves the room: Disconnected. [11:51:55] Someone == Stefan Wegner [11:52:03] Need to define some other way to document conformance. [11:52:52] Jakov Stein: What I wanted to say has been said... bit exactness reqts came about because of 20-30 percent increase in MIPS for bit-exactness... haven't heard of perf. restriction here.... [11:52:52] tomkri leaves the room [11:53:14] Jakov Stein: we can forget about bit-exactness... as long as we have another way to document conformance. [11:53:24] Michael: no requirement for bit exactness? [11:53:30] Cullen: Any objections? [11:53:39] No objections heard. [11:53:40] Please comment on mic: one possible other use for bit exactness is for ASR [11:54:05] ASR == automatic speech recognition use case [11:54:19] but I will bring up on list if necessary [11:54:46] Stefan Wenger: It is desirable... an area of fundamental research. [11:54:54] Barry Leiba is bringing it to the mic, Dan. [11:55:02] thx resnick [11:55:09] tomkri joins the room [11:55:55] jean-marc: we are talking about combining two channels, taking redundancy out, not a particular implementation... in current codec there are some benefits... this is not ms encoding, that is different. [11:56:37] gregory joins the room [11:56:50] thx Barry [11:57:24] Stefan Winter: gains of 35% compared with sending two channels... worth leveraging. [11:57:40] Michael: Desired joint stereo mode if inexpensive computationally? [11:57:51] Cullen: does anyone object to desired? Not mandatory. [11:58:01] Cullen: Let's make it "highly desired". OK? [11:58:28] Jean-Marc: can look at codec with and without, with joint stero vs. dual mono.... should become clear if it is a good idea or not. [11:58:50] Jonathan: this reqt. is hard to evaluate in the absence of test data.... [11:59:34] Stefan Wenger: chicken-egg problem... traditionally we set reqts, then select based on that... we are using dialectic process here.... between proponents and reqts work. It is unclean! [12:00:06] Stefan Wenger: it is simple to come to a decision based on stereo... not just about a particular implementation. [12:00:54] Cullen: I agree with that... here is what I am hearing.... this is not mandatory regardless of cost.... it is clearly something that people are excited about, could generate 30 percent bitrate savings... not irrelevant, not mandatory... desirable. [12:01:10] Cullen: it is in between the extreme... if we have consensus, we should put it in the reqts. [12:01:19] Joe Hildebrand joins the room [12:01:53] Jakov Stein: wanted "highly desirable" because if it is stero sound it should be mandatory... but if most the time we're in mono, it isn't a requirement.... [12:02:10] Christian: joint stereo should be optional.... [12:02:15] let's move on [12:03:10] Stefan: logitech has stereo... we are going in this direction even for interactive services... to be future proof, we should do stereo in an efficient way. [12:03:24] we already decided at the very beginning that this was primarily a voice codec, but may be used for music being sent over a "call" channel, i.e. on hold music, or rich media "calls". So it's a nice to have [12:03:25] Michael: move to stereo is happening. [12:03:55] if we want to focus on a codec optimized for stereo, we already agreed we would/could do a separate codec for that in future [12:04:06] Jonathan Rosenberg: We'll put in "highly desirable" as the consensus. [12:05:09] Joanthan Lennox: Is the stereo actually working as stereo? It is a hard problem to demonstrate.... [12:05:38] Michael: Shallwe offer "low delay" mode? proposal is DESIRED. [12:06:30] Michael: lookahead + frame size = total delay. [12:06:46] Cullen: No objections to things that low... but have no need for it either. [12:07:07] Cullen: made as an individual contributor (not chair). [12:07:28] Jean-Marc: current implemetation supports 5 ms... a few applications need that. [12:07:40] Michael: Objections to DESIRED? None. [12:07:56] Packet loss concealment [12:08:33] Required? Yes. perf goals for PLC? informative... upgradeable? Yes. Integration with VAD/CNG? Diesred. [12:09:31] Stefan Wenger: packet loss concealment... would it mean that codec spec would normatively define a way to deal with lost packets? In that case, I'd suggest it is a bad idea. [12:09:58] Stefan wenger: not precise enough to comment on it. [12:10:20] Michael: As part of package, put in a nominal mechanism... but could do something else. [12:10:39] Jean-Marc: reference needs to provide one way of doing it... not a normative way. [12:11:16] Michael: packet loss scenarios.... typically 3-5 percent, bursty or not... no goals yet. [12:11:43] Christan: PLC term not complete enough... needs to cover packet loss and other adaptations.... [12:12:17] resnick leaves the room [12:12:20] Stefan Wenger: would prefer to talk about what a standards track document will provide at the end... don't care about reference implementation... we're not managing an open source project, we're writing a spec! [12:12:49] Stefan Wenger: an informative section on how PLC can be implemented would make more sense. [12:13:19] Cullen: Reqts doc says we have an informational section on implementation of PLC. No objections to that approach to reqts? [12:14:02] Michael: No issue in integrating with VAD/CNG (DESIRED). [12:14:17] Next on agenda: Christian: rate adaptation. [12:14:48] Problem statement: IIAC is likely to have many parameters. [12:15:10] Problem: when to set which codec parameter how? [12:16:29] Cullen: need packet loss concealment on your powerpoint slides (heh). [12:16:50] Fabricio leaves the room [12:17:39] ITU defines "quality of experience" [12:17:54] includes complete end-to-end system... [12:17:56] gregory leaves the room [12:18:28] One idea was to use DCCP vs. UDP..... [12:20:58] JonathanLennox leaves the room [12:22:48] Sumamry: IIAC + RTP + DCCP is a useful combination.... [12:23:02] but need protocl support for QoE control loop... [12:24:03] Vendor specific optimizations possible... [12:24:08] Push to talk mode helps. [12:24:58] Colin Perkins: I like DCCP... but struggle to see relevance to this WG.... [12:25:17] Colin Perkins: can't mandate transports.... it could run over TCP, SCTP, UDP, DCCP, etc. [12:25:25] Jonathan: or ATM... no dispute on this. [12:28:13] I believe Christian is arguing that there may optimizations that should be specified in the codec design for when DCCP is used [12:28:51] not that the CODEC can only be used with that one transport mechanism [12:29:48] Michael: we need to perform subjective testing.... [12:30:35] which slide # did we start at ? [12:30:42] Recommendation: testing workflow. Phase: development. [12:30:49] It says slide 20. [12:30:53] mlm.michael.miller joins the room [12:31:20] thx [12:31:58] We want to come up with tools to do bug finding or conformance testing methology. [12:32:06] tomkri leaves the room: Replaced by new connection [12:32:12] tomkri joins the room [12:32:42] Codec "Test Houses". Ask for 3-5 volunteer companies... agreed to provide recommended testing signal chain and audio environment. Expected $5K- $10K budget. [12:33:43] Summary: comprehensive testing of codec is challenging. [12:33:49] gregory joins the room [12:35:01] Michael: don't need booth, need quiet environment.... [12:36:08] Stefan Wenger: I like 3-5 companies.... this is to be tested by someone who doesn't like your work.... [12:36:38] Eric Rescorla: you said it wasn't to be used as a threshold for acceptance... how do you imagine consuming the data and doing something useful with it? [12:37:18] hta joins the room [12:37:18] Jonathan: proposal is to not make it a gating factor for IESG... but we include it in a PROTO writeup... further tests can happened... proposed to draft bar gets higher when going to draft, but not a requriement for PS. [12:37:29] Eric: If this never gets done it's fine for PS? [12:37:42] Jonathan: we won't wait for this to bring document to IESG for PS. [12:38:08] Jean-Marc: who is interested in doing testing? [12:38:39] gregory leaves the room [12:38:39] Marc: who wants to test this codec? 5? 6? [12:38:41] resnick joins the room [12:38:48] Michael: send contact info to the list. [12:39:28] Stefan Wenger: Don't you need environment used in the labs to be the same? [12:40:16] Stefan Wenger: There should be a certain minimum commitment from a number of companies, not individuals.... if it cannot be reached, then we have a seroius problem. [12:40:27] gregory joins the room [12:40:29] Michael: we don't have a serious problem because people are raising their hands.... [12:40:58] FWIW, I'm just passing on this request to my contacts in the UK radio industry. Hopefully suffieicnt people there will be able to offer expertise. [12:41:11] Stefan Wenger: I agree that 3-5 companies is the level of commitment that is necessary... the minimum is that the WG at large can field an investment of $30K. [12:41:55] Cullen Jennings: I already saw 4 companies hands go up... suspect that WG will want to see results from companies they trust.... [12:42:45] Cullen: what is serious testing? Codec will be used with inexpensive microphones in noisy environments... [12:43:37] Michael Knapp: people can test with $3 airplane headphones in an airplane.. but that's not how you evaluate it. [12:44:09] Chris from Mozilla: we use Amazon's mechnical turk you can thousands of responses. [12:44:51] Jean-Marc: multiple companies in very controlled environments... you will have 3-5 sets of identical results you could have gotten with a single test. More useful to have a range of conditions. [12:44:58] Cullen: We're asking for both. [12:44:59] Joe Hildebrand leaves the room: Disconnected. [12:48:31] (someone) are you suggesting this be done double blind/ [12:48:57] (someone else): Yes, full double blind. [12:49:19] (someone): Then why do you care if someone hates your codec? they won't know which one is being tested! [12:50:19] Stefan Wenger: Don't care if codec is accepted by open source community... I'm interested in serious company acceptance.... they will want confidence that tests have been performed to a confidence level where the company doesn't have to redo tests themselves... or they will have to redo them. [12:52:19] Kun-Voss: Rigorous testing as done by ITU has two goals: needs to be blind in all the right ways and we agree that's what we want.... they don't want to do a listening test that requires hundreds of listeners.... if we can get many hundreds, then we don't have to be as rigorous as ITU... noise can be filtered by large number of results. [12:52:47] Jonathan (as an individual): I couldn't agree more. Skype as done testing with actual users... scale makes up for lack of an ISO booth measurement.... [12:54:11] Jean-Marc: Testing can be done after going to IESG. Companies that want to adopt codecs still need to do their own testing... some companies have had surprises with G.729 in their particular environments.... [12:55:03] Cullen: Guidelines doc is not done (issues on this topic as well as others). Want to hum for adoption as a WG work item. Doesn't mean you agree, but it's the one we will start with. [12:55:17] Cullen: Anyone to the mike? [12:55:59] JonathanLennox joins the room [12:56:33] Stefan Wenger: I apologize for being a pain today... we didn't adopt the doc because there was major disagreement over some aspects.... nothing has happened in 4 months in terms of development does not make disagreement go away. They are still there... doc is under-specifying and is totally biased in a direction I don't like. So I am *not* happy with adopting it, just because discussions have calmed down. Just because noone has objected doesn't mean that there isn't an objection. [12:56:46] Stefan Wenger: this is not the right way to move the WG or this document. [12:57:11] Christian: I had a view of this doc... most parts were outdated... it needs to updated.... there have been few discussions.... [12:57:25] Michael: we will disagree about ways of testing. [12:58:03] Cullen (Huddles with Jonathan). [12:59:31] Cullen: I agree with Stefan's comments... arguments stopped but weren't resolved... we won't take a hum on this now... we need this doc to be updated soon... we need people to express their objections that prevent adoption on the list and then have it out before the next meeting.... if this is the only doc on the table for this milestone we will be heavily leaning toward a hum. [12:59:55] Robert Sparks: would it be reasonable to set a deadline for a potential drafts for this milestone? [13:00:02] Cullen: October 15, 2010 as a potential date? [13:00:22] Jonathan: 6 weeks is fine. [13:01:07] Stefan Wenger: we should see the revv'd document out... then give people a few weeks to come up with change requests.... basic structure of doc is right... but content is not right in my opinion. One way is to take stupid part out and leave structure intact. [13:01:09] RjS leaves the room [13:01:27] Cullen: Can we get a document rev in two weeks? [13:02:01] Cullen: We will set a deadline of September 15 (correction). [13:02:44] Jonathan: don't have to have complete answer... thoughts on a process are ok... doesn't have to be complete for everything else... goal is to get alternatives on the table... want bar to be set low. [13:03:12] Jean-Marc: early tonite I will send to list the most up to date version... we can discuss based on that version. [13:03:31] Jonathan: Excellent! Update shortly... people put in comments, alternate proposals, everything on table by September 15, 2010. [13:03:33] Barry Leiba leaves the room [13:03:38] juberti leaves the room: Computer went to sleep [13:03:41] Cullen: I think we will close the meeting! [13:03:41] YJS leaves the room [13:03:50] JonathanLennox leaves the room: Computer went to sleep [13:03:50] Jonathan: Meeting adjourned. [13:04:00] 3:05 PM. [13:04:05] tomkri leaves the room [13:04:06] mlm.michael.miller leaves the room [13:04:11] dwd leaves the room: offline [13:04:19] suzukisn leaves the room [13:04:28] christopher.blizzard leaves the room [13:04:43] sorenskakjensen leaves the room [13:05:58] as0-d91k leaves the room [13:09:54] burn leaves the room [13:10:41] hta leaves the room [13:14:30] RjS joins the room [13:15:45] RjS leaves the room [13:18:54] Cullen Jennings leaves the room [13:19:16] Cullen Jennings joins the room [13:21:21] Cullen Jennings leaves the room [13:23:29] Minute-Taker leaves the room [13:23:30] resnick leaves the room [13:24:31] gregory leaves the room [13:25:43] hta joins the room [13:36:00] hta leaves the room [13:36:02] Fabricio joins the room [13:36:57] Fabricio leaves the room [13:51:10] Cullen Jennings joins the room [13:51:35] Minute-Taker joins the room [13:53:43] Cullen Jennings leaves the room [14:01:24] Joe Hildebrand joins the room [14:01:27] Joe Hildebrand leaves the room [14:30:58] kvs78 joins the room [14:59:10] gregory joins the room [15:01:05] kvs78 leaves the room [15:02:59] gregory leaves the room [15:22:29] Minute-Taker leaves the room [15:56:19] Minute-Taker joins the room [17:22:20] tonyhansen leaves the room [17:34:01] Minute-Taker leaves the room [19:51:55] Minute-Taker joins the room [21:04:03] Minute-Taker leaves the room